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A ROUTING NETWORK 
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Cross-Reference to Related Applications 

This application contains subject matter which is 
related to the subject matter of the following United States 
patent applications, which are assigned to the same assignee 
of this application. Each of the below-listed applications 
is hereby incorporated herein by reference: 

''Routing Messages Within A Network Using The Data 
Content Of The Message," by Chandra et al., filed 
November 20, 1997, serial no. 08/975,303; 



''Message Logging For Reliable Multicasting Across 
A Routing Network," by Banavar et al . , co-filed 
herewith, serial no. , (attorney docket no. 

15 Y0998-525) ; 



"Message Sequencing For Ordered Multicasting Of A 
Message Across A Routing Network," by Banavar et al,, 
co-filed herewith, serial no. , (attorney 

docket no. Y0998-526) ; and 

"Non-Disruptive Reconfiguration Of A 
Publish/Subscribe System," by Miller et al . , co-filed 
herewith, serial no. , (attorney docket no. 

Y0999-124) . 
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Technical Field 



This invention relates in general to reconfiguration of 
a routing network and, in particular, to quiescently 
reconfiguring a routing network having one or more special 
nodes for facilitating reliable delivery or ordered delivery 
of messages within the routing network using logging or 
sequencing techniques employing the special node(s). 

Background of the Invention 

Many network environments enable messages to be 
forwarded from one site within the network to one or more 
other sites using a multicast protocol. Typical multicast 
protocols send messages from one site to one or more other 
sites based on information stored within a message header. 
That is, each message has two components: the message 
header, which includes the routing information, including 
destination addresses or a predefined group name that is 
associated with a fixed list of destinations, and a data 
content, which is the data of the message. The routing 
information is read from the message header and is used to 
send the data content of the message to the specified 
destinations . 

One example of a system that conventionally includes 
such a network environment is a publish/subscribe system. 
In publish/subscribe systems, publishers post messages and 
subscribers independently specify categories of messages in 
which they are interested. The system takes the posted 
messages and includes in each message header the destination 
information of those subscribers indicating interest in the 
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particular message. The system then uses the destination 
information in the message to forward the message through 
the network to the appropriate subscribers. 

More particularly, a publish/subscribe system includes 
5 a network of message routers (or simply routers or nodes) 
connected via links in an arbitrary graph topology. A 
number of clients connect to the periphery of this router 
network and either publish or subscribe to messages. A 
message includes of a number of attributes, which are name- 
10 value pairs. A subscription specifies a predicate on the 
attributes of messages. 

The problem addressed by the present application is how 
to handle changes of topology in such a routing network. 
The need for changing topology arises from a number of 

15 conditions, including: (1) the need for system maintainers 
to move routers on- and off-line; (2) the growth of the 
network; and (3) changes in link speed that alter the 
optimum spanning trees of the network; etc. An assumption 
is made that a configuration manager has made a decision to 

20 effect a particular topology change (i.e., a 

reconfiguration) . The present invention is directed to a 
technique by which nodes of the router network execute a 
reconfiguration decision, eventually resulting in a new 
state of the network in which messages are forwarded using a 

25 new spanning tree. 

Disclosure of the Invention 

Briefly summarized, the present invention comprises in 
one aspect a method for reconfiguring a routing network. 
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The method includes: quiescing the routing network to 
preserve a first-in first-out ordering of data messages 
within the routing network; and reconfiguring the quiesced 
routing network- As an enhancement, the method further 
5 includes the quiescing of control messages within the 

routing network prior to the reconfiguring, and the routing 
network comprises a special network having at least one 
logging node or a sequencing node wherein messages are 
either logged to persistent storage for reliable routing or 
10 sequenced with other messages received by the network for 
ordered delivery. 

In another aspect, the invention comprises a system for 
reconfiguring a routing network. The system includes means 
for quiescing the routing network to preserve a first-in 
15 first-out ordering of data messages within the routing 

network, and means for thereafter reconfiguring the quiesced 
routing network. 

In still another aspect, a system for reconfiguring a 
publish/subscribe system is provided. This reconfiguration 

20 system includes a routing network adapted to receive 

published messages for forwarding to subscribers. The 
network is further adapted to quiesce data messages and 
control messages within the network in response to a 
reconfigure command to preserve a first-in first-out 

25 ordering of data messages and control messages within the 

routing network, and to then reconfigure the routing network 
once data messages and control messages have been quiesced. 

In a further aspect, the invention comprises an article 
of manufacture which includes at least one computer usable 
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medium having computer readable program code means embodied 
therein for effecting reconfiguring of a routing network. 
The computer readable program code means in the article of 
manufacture includes: computer readable program code means 
for causing a computer to effect quiescing the routing 
network to preserve a first-in first-out ordering of data 
messages within the routing network; and computer readable 
program code means for causing a computer to effect 
reconfiguring the quiesced routing network. 

To restate^ provided herein is a technique for 
quiescent reconfiguration of a routing network of a 
publish/subscribe system without loss of messages. 
Reconfiguration is dynamic in that it is accomplished 
without shutting down the network, i.e., the execution of a 
reconfiguration is not directly visible to either publishers 
or subscribers. The publishers and subscribers continue to 
publish and receive messages as if no reconfiguration is 
taking place or has taken place. As used herein, quiescent 
means that it is only necessary to temporarily suspend 
message traffic within the routing network before 
reconfiguring the network and resuming message traffic. The 
quiescent reconfiguration technique presented has particular 
application in large, continuously available broker networks 
in which change and evolution are inevitable and service 
disruption is not tolerable. Although principally described 
below in connection with a content-based routing network, 
the reconfiguration approach can be applied to other routing 
systems, e.g., subject-based routing systems. 
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Brief Description of the Drawings 

The above-described objects, advantages and features of 
the present invention, as well as others, will be more 
readily understood from the following detailed description 
5 of certain preferred embodiments of the invention^ when 

considered in conjunction with the accompanying drawings in 
which : 

Fig, 1 depicts one example of a distributed network 
incorporating and using the quiescent reconfiguration 
10 capability of the present inventions- 
Fig. 2 depicts one example of a spanning tree used in 
accordance with the principles of the present inventions- 
Fig. 3 depicts one embodiment of a distributed router 
network, configured with resilient or ordered routing 
15 capability, which is to undergo quiescent reconfiguration in 
accordance with the present invention; 

Fig. 4 depicts one embodiment of a router for use in 
accordance with the principles of the present invention in a 
router network such as depicted in Fig. 3; 

20 Fig. 5 depicts one example of an information table for 

routers of a network employing quiescent reconfiguration in 
accordance with the principles of the present inventions- 
Fig, 6 is a flowchart of one embodiment of CONFIGURE 
message processing in accordance with the principles of the 

25 present invention; 
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Fig. 7 is a flowchart of one embodiment of QUIESCE 
message processing in accordance with the principles of the 
present invention; 

Fig, 8 is a flowchart of one embodiment of 
QUIESCE_ACKNOWLEDGE message processing in accordance with 
the principles of the present invention; 

Fig. 9 is a flowchart of one embodiment of 
QUIESCE_SPECIAL message processing in accordance with the 
principles of the present invention; 

Fig. 10 is a flowchart of one embodiment of 
QUIESCE_SPECIAL_ACKNOWLEDGE message processing in accordance 
with the principles of the present invention; 

Fig. 11 is a flowchart of one embodiment of 
QUIESCE_COMPLETE message processing in accordance with the 
principles of the present invention; and 

Fig. 12 is a flowchart of one embodiment for processing 
published messages in a routing network undergoing quiescent 
reconfiguration in accordance with the principles of the 
present invention . 

Best Mode for Carrying Out the Invention 

In one aspect, the present invention can be employed in 
a system using the routing capability disclosed in the 
above-incorporated application wherein messages are routed 
in a network based on the content of the data within the 
message. With data-content routing, a message does not need 



to include destination information, such as destination 
addresses or a group destination name. Instead, data within 
the message is used to traverse a data structure to 
determine the link or links over which the message is to be 
5 forwarded in order to reach the consumers (subscribers or 
clients) interested in the message. 

More particular to the present invention, a quiescent 
reconfiguration technique is provided for, for example, such 
a content-based multicasting network. Further, the 

10 quiescent reconfiguration technique presented herein may be 
used in combination with, or separately from, the ''reliable 
routing capability" and the ''total ordering capability'' 
described in the above-incorporated, co-filed patent 
applications. Whether used in combination with reliable 

15 routing or ordered delivery, or separately therefrom, 

quiescent reconfiguration in accordance with the principles 
of this invention may also be used with the above-described 
content-based routing network. However, those skilled in 
the art will recognize that the reconfiguration technique 

20 presented below is equally applicable to other routing 
systems, such as a subject-based routing system. One 
assumption underlying the reconfiguration technique is that 
links within the routing network propagate messages in a 
reliable first-in first-out manner. 

25 One example of a distributed network 100 incorporating 

and using the quiescent reconfiguration capability of the 
present invention is depicted in Fig. 1 and described in 
detail below. Network 100 includes, for instance, a 
plurality of computing units 102 coupled to one another by 

30 links 104. 
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Each link couples two computing units in the network, 
and each computing unit may have any number of links 
connected to it. Each link is bidirectional, i.e., a 
computing unit may send and receive messages on the link. 
5 Each link is also connection oriented, i.e., the computer 
units at the ends of the link are notified when the link 
fails. Furthermore, each link propagates messages in a 
first-in first-out manner. Each computing unit in the 
network is either a client computer (represented by the 

10 smaller ovals, such as those having addresses 101a, 101b) , 
meaning that it either originates messages or has requested 
to receive messages whose content satisfies certain 
properties; or it is a router computer ^ (represented by the 
larger ovals, such as 102a, 102b), meaning that it forwards 

15 messages received on one network link onto other links on 
the way to the client computer (s). -The clients are 
collectively referred to herein as clients 101 and the 
routers are collectively referred to as routers 102. (For 
purposes of this discussion, if a single computing unit 

20 serves both as a router and as a client, these two separate 
functions will be considered as two computing units 
connected by a link.) 

Each computing unit can be any type of computing unit 
that can be included in a network. For example, it can be 

25 an RS/6000 computing node or any other of various types of 
computing nodes, processors, computers or systems. The 
network can also include different types of computing units 
coupled to one another via the links. The links include, 
for instance, TCP connections over IP links, as only one 

30 example. 
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Distributed network 100 can be included in various 
systems that require the passing of messages or data. These 
systems include, for instance, the Internet or 
publish/subscribe systems (such as the above-summarized 
5 content-based messaging system) . 

Content-based publish/subscribe systems improve the 
degree of decoupling between publishers and subscribers. In 
content-based publish/subscribe systems, subscriptions are 
specified in terms of predicates on the posted data, rather 

10 than in terms of subject identifiers supplied by the 

publisher. One example of a content-based publish/subscribe 
system is described in co-pending U.S. Patent Application 
Serial No. 08/975,280, entitled "Method And System For 
Matching Consumers To Events," Astley et al., which is 

15 hereby incorporated herein by reference in its entirety. 
The examples described herein are with reference to a 
content-based subscription system. However, these are only 
examples. The present invention can be employed with other 
types of systems without departing from the spirit of the 

20 present invention. 

In one embodiment of the invention, each router 102 of 
network 100 (Fig. 1) has associated therewith a spanning 
tree, which lays out the best path (according to some 
criterion, such as latency) from the router to each client 
25 101. In this embodiment, it is assumed that routers agree 
on a common criterion for measuring distance between nodes 
in the network. There may in fact be multiple spanning 
trees. For example, alternative spanning trees may specify 
either backup routes, or peak load routes. Herein, it is 
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assumed that one spanning tree is initially in effect for 
the routing of any particular message. 

One example of a spanning tree, which is associated 
with a router is depicted in Fig. 2. As shown in Fig. 2, 
5 there is a path from a router 102a to every other node in 
the spanning tree. A message to be routed from router 102a 
to one or more of the other nodes is routed via one or more 
of the links associated with router 102a, i.e., links 1-3. 
For example, if a message is to be routed from node 102a to 
10 node 101c, then link 2 is used. As a further example, if a 
message is to be forwarded toward client 101a, then link 3 
is used. 

One embodiment for building a spanning tree from a 
network topology, that is an arbitrary graph, is described 
15 in detail in Introduction to Algorithms , by Gormen, 

Leiserson, Rivert, Chapter 24, pp 498-513, Published by MIT 
Press (1990) , which is hereby incorporated herein by 
reference in its entirety. 

In addition to the spanning tree associated with each 
20 router, each router has a routing table. The routing table 
includes an entry for each client computer in the network. 
Each entry of the routing table associates a client address 
with the identifier of the network link constituting the 
next segment on the path in the spanning tree from the 
25 router to the client. For a router with d network links, 
each such link identifier is an integer between 1 and d. 
For instance, the client having address 101a has a 
corresponding link identifier of 3 (see Fig. 2) . 
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Each routing table is constructed via information from 
the network topology (e.g., the client addresses) and hence 
from the corresponding spanning tree (e.g., the link 
identifiers), in a known manner. 

5 

By way of example, Fig. 3 depicts one embodiment of a 
publish/subscribe system, generally denoted 300, to employ 
quiescent reconfiguration in accordance with the principles 
of the present invention. System 300 includes a network of 
10 routers 302 connected via links 303 in an arbitrary graph 

topology. A number of clients connected to the periphery of 
this router network either publish messages or subscribe to 
messages. Those clients publishing messages comprise 
JS publishers 304, while those clients subscribing to messages 

jy 15 comprise subscribers 306. The router network is responsible 
nj for routing messages from a publisher 304 to interested 

^ subscribers 306 based on matching messages to subscription 

predicates. 

y Again, this protocol (i.e., content-based routing) is 

^ 20 described in detail in the initially-incorporated co-pending 
41 patent application entitled ''Routing Messages Within A 

'^'^ Network Using The Data Content Of The Message." As noted, 

from each router node at which a publisher is present, the 
system computes and stores a spanning tree to reach every 
25 other node in the network. All published messages from the 
publisher follow the paths in that spanning tree, with each 
router node performing enough matching to determine which of 
its child routers should receive the message; 

Within a publish/subscribe system, the present 
30 invention can be employed, in one example, with a network 



Y0998-527 



achieving reliability of routed messages by saving messages 
to persistent storage within the network (and then 
retrieving and redelivering the message whenever there is a 
failure in the network) . Such a reliable routing network is 
5 described in the above-incorporated, co-filed patent 
application entitled "'Message Logging For Reliable 
Multicasting Across A Routing Network." In accordance with 
that invention, publishers and subscribers that need 
reliability of messages may specify a quality of service 

10 parameter, e.g., "'uniform delivery". Uniform delivery is 
provided for ensuring delivery of a message to all active 
subscribers notwithstanding failure in the network, e.g., 
the routers, or the links. One or more special routers in 
the router network 300 are designated logging node(s) or 

15 logger (s) 310 and support the^ ability to log messages to 
stable storage. When there is at least one subscriber 
needing logging, the routing algorithm ensures that messages 
are routed to the logger. 

Alternatively, or in further combination, the present 
20 invention can be employed in another example with a network 
achieving ordering of routed messages by sequencing messages 
within the router network itself through assignment of a 
sequence number at a designated sequencing node of the 
network. Publishers and subscribers that need ordering of 
25 messages may specify a quality of service parameter called 
""totally ordered". Total ordering is provided for ensuring 
ordered delivery of a message to all active subscribers. 
Such message sequencing within the router network is 
described in the above-incorporated, co-filed application 
30 entitled ""Message Sequencing For Ordered Multicasting Of A 
Message Across A Routing Network." When employing 
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sequencing, a special router 310 in the router network 300 
is designated a sequencing node or sequencer. Node 310 
supports the ability to sequence messages received into the 
routing network. When there is at least one subscriber 
5 needing total ordering, the routing algorithm guarantees 
that messages are routed to the sequencer. 

To summarize, quiescent reconfiguration pursuant to the 
present invention can be employed in a router network of a 
publish/subscribe system which utilizes at least one special 
10 node to facilitate either, or both, logging of messages or 
sequencing of messages with the network itself. 

Fig. 4 depicts one embodiment of a router node for use 
in accordance with the principles of the present invention. 
When a message arrives at router 400 it is stored into a 

15 message table 402 using a unique message identifier. 

Information stored includes a source node identification as 
well as the neighboring nodes to which the message is to be 
forwarded. These neighboring nodes are calculated by a 
content routing computation component 404 after the message 

20 has been received. Computation component 404 takes the 
message and, based upon stored subscriptions and the 
spanning tree, returns a set of destinations or links upon 
which the message should be forwarded. Again, in one 
embodiment, this computation is content dependent and can be 

25 accomplished as described in the above-incorporated co- 
pending application. However, this component can be 
implemented in other ways, e.g., subject-based routing. In 
one embodiment, to facilitate reliable routing of messages, 
router 400 would also include a reliable routing component 

30 406 and logging acknowledgment (LACK) received table 408 and 
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LACK send table 409, which are described in the above- 
incorporated, co-filed application entitled ''Message Logging 
For Reliable Multicasting Across A Routing Network," 

If ordered routing of messages is employed, then router 
5 400 would also include recovery data, including a latest 

sequence number received and linked node tables (not shown) 
such as described in the above-incorporated, co-filed 
application entitled ''Message Sequencing For Ordered 
Multicasting Of A Message Across A Routing Network." Again, 
10 the quiescent reconfiguration approach of this invention can 
be employed in a routing network of a publish/subscribe 
system employing data content messaging, reliable routing of 
messages, or sequencing of messages, either individually or 
nj in combination. 

15 Quiescent reconfiguration requires the entire routing 

^ network to be quiesced before the network is reconfigured 

and network operations are resumed. Since the network is 
y quiesced before it is reconfigured, first-in first-out 

^ (FIFO) ordering is preserved between all nodes. In 

^ 20 accordance with this invention, once quiesced, a node may 
'^^ not publish any new messages from one or more clients 

thereof. However, a quiesced node- must continue to route 

any data messages or control messages which it receives from 

other nodes of the network. 

25 In one embodiment, the following messages are used to 

quiesce a network in accordance with the present invention. 

• QUIESCE - sent by the root of a configuration 

change to initiate quiescence of data messages. 
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• CONFIGURE - sent by a configuration manager and 
nodes in the network to change a spanning tree 
used to route messages in the network. 
QUIESCE_ACK - sent by a node to acknowledge 
5 receipt of a QUIESCE message. Once a QUIESCE_ACK 

has been sent^ a node cannot forward messages from 
clients into the routing network any new data 
messages for transmission across the network. 
However, it must continue to route all data and 
10 control messages sent by other nodes in the 

network. 

QUIESCE^SPECIAL - sent by a root to quiesce a 
special node, such as a logging node or a 
sequencing node. The special node responds with a 
15 QUIESCE_SPECIAL_ACK message. 

QUIESCE_COMPLETE - sent by the root to indicate 
that all nodes have been quiesced and the new 
configuration may be resumed. 



An overview of the algorithm for quiescing and 
20 reconfiguring a broker network is given next. First, a 

decision is made to reconfigure the broker network. This 
decision is outside of the scope of this description. This 
description assumes that a CONFIGURE message is communicated 
to a node in the network which become^s the root of the 
25 configuration change. For example, a Configuration Manager 
(described below) under control of some administrator might 
calculate a new configuration for the network and send a 
CONFIGURE message to the root. Parent and child nodes are 
defined in relationship to this root node. 
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In one embodiment, quiescent reconfiguration in 
accordance with this invention can then proceed in three 
phases. First, data messages within the network are 
quiesced. Second, control messages ( protocol messages used 
5 by special nodes, i.e., loggers and sequencers, e.g., 

LACK/SACK messages) are quiesced. Third, the network is 
reconfigured. These phases are summarized below. 

Data Message Quiescence 

The root node initiates quiescence by sending a QUIESCE 
10 message to all child nodes thereof using the existing 

configuration. When a node receives a QUIESCE message, the 
node begins queueing new data messages received from its 
clients, forwards the QUIESCE message to all child nodes, 
and sends a QUIESCE_ACK to its parent. QUIESCE_ACK messages 
15 may be coalesced at each level (i.e., a parent may wait to 

receive QUIESCE_ACKs from all of its children before sending 
its own QUIESCE_ACK to its parent), or all QUIESCE_ACK 
messages may be forwarded to the root node. Once quiesced, 
a node does not forward any new data messages from clients 
20 thereof. However, the node continues to forward data and 
control messages received from neighboring nodes (parents 
and children) . Also, a logging/sequencing node continues to 
produce LACK/SACK messages in response to data messages 
which it receives. 

25 Once the root node has received^ a >QUIESCE_ACK message 

from all nodes, from the root's perspective, all data 
messages have been quiesced, and there can be no data 
messages flowing through the network towards the root. 
However, there may be data messages flowing away from the 
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root node. Also, there may be SACK and LACK messages 
flowing towards and away from the root node. 

Control Message Quiescence 

The root node next quiesces LACK/SACK messages in the 
network. Using the same quiesce technique outlined above, 
the root node sends QUIESCE__SPECIAL messages to any children 
sequencers and loggers. Note that if the root node is the 
only special node in the network, the control message 
quiescence phase is not necessary. ' When a logger/sequencer 
receives a QUIESCE_SPECIAL message, the node will not 
receive any new data messages on the current configuration. 
When all LACK/SACK messages to any previous data messages 
have been sent, the logging/sequencing node sends a 
QUIESCE_SPECIAL_ACK to the root node. After a 
logger/sequencer node has been quiesced, the node continues 
to forward any control messages received from neighboring 
nodes (parent and children) . 

Once the root has received a QUIESCE_SPECIAL_ACK 
message from all logger/sequencer nodes, then from the 
root's perspective, all control messages have been quiesced 
and there are no control messages flowing through the 
network towards the root node. However, there may still be 
data and control messages flowing away from the root node. 

Reconfiguration 

The root next informs all nodes that the network has 
been quiesced and proceeds to reconfigure the network. 
First, the root node sends a QUIESCE COMPLETE to all nodes 
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using the existing configuration. Because links between 
nodes are FIFO, the QUIESCE__COMPLETE message is guaranteed 
to be the last message in the existing configuration, i.e., 
no data or control message can follow the QUIESCE_COMPLETE 
5 message. Next, the root of the new configuration (if 

different from the old root, the old root would have to send 
a message to the new root), sends a CONFIGURE message to all 
nodes using the new configuration. When a node receives a 
CONFIGURE message, the node forwards the message to all of 

10 its downstream nodes in the new configuration. If a node 
receives a CONFIGURE message before it has received a 
QUIESCE_COMPLETE message on the previous configuration, the 
node must wait for the QUIESCE_COMPLETE before processing 
any new messages received on the new configuration. Waiting 

15 for the QUIESCE_COMPLETE message insures that all data and 
control messages have been received on the old 
configuration. Once a QUIESCE_COMPLETE message is received, 
any resources allocated for the previous configuration are 
no longer needed and may be freed. 

20 One detailed embodiment of quiescent reconfiguration in 

accordance with the present invention is described below 
with reference to Figs. 5-12. Fig. 5 depicts one example of 
additional information employed at the routers of a network 
to use quiescent reconfiguration in accordance with this 

25 invention. For example, the table of Fig. 5 could be 
associated with or incorporated within the quiescent 
reconfiguration component 405 of the router of Fig, 4. This 
table includes (in one example) entries for five variables, 
labeled RECONFIG_ROOT, QUIESCING, CONFIGURE_RECEIVED, 

30 QUIESCE_MSG_QUEUE, and QUIESCING CHILDREN. Each of these 
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variables is described below in connection with the 
flowchart process of Figs, 6-12. 

Fig. 6 depicts one embodiment for processing a 
CONFIGURE message received, for example, from a 
5 configuration manager. As noted above, an assumption is 

made that the configuration manager independently makes the 
decision to effect a particular topology change, i.e., a 
reconfiguration. A configuration manager is a ^'network 
system service'' that is responsible for maintaining 

10 information about the structure, function and status of a 
network system. This includes information such as the 
network topology, nodes with certain properties (such as 
loggers, sequencers), quality of service offered by nodes in 
the network, etc. A system's configuration may be 

15 statically fixed or dynamically changing. A configuration 
manager may be centralized or distributed. Network system 
management products such as IBM's Tivoli TME-10 contain 
configuration management components. Again, from the 
perspective of the current invention, a CONFIGURE message is 

20 assumed to be received from the configuration manager. The 
present invention is directed to a technique by which 
reconfiguration is transparent to the publishers and 
subscribers of the network, particularly where the network 
employs a special logging or sequencing node as described 

25 above. 

Referring to Fig. 6, processing commences with receipt 
of a CONFIGURE message 600 within the network. With receipt 
of this message, the node determines whether it is the root 
of the reconfiguration 605. If ''no", then the CONFIGURE 
30 message is forwarded to all children nodes in the new 
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configuration 610, and the node determines whether the 
QUIESCING state is true 615. If ^^yes", then the 
CONFIGURE_RECEIVED indicator is set 620. Otherwise, the 
reconfiguration is complete and the node begins forwarding 
5 messages from its message queue (QUIESCE_MSG_QUEUE ) using 
the new spanning tree 630. 

Returning to inquiry 605, if the node is the root of 
the reconfiguration, then the node's RECONFIGURE_ROOT 
indicator is set 640, as well as the QUIESCING indicator 
10 650. A QUIESCE message is sent to all children in the 
current configuration 660 and the QUIESCING_CHILDREN 
indicator is set to the number of children nodes in the 
current configuration 670. If the QUIESCING_CHILDREN count 
W is zero 680, then processing jumps to step 850 of Fig. 8 

nj 15 (described below) . 

U Fig. 7 is a flowchart of one embodiment for QUIESCE 

JU message processing in accordance with the present invention. 

yj When a node receives a QUIESCE message 700, the node's 

QUIESCING indicator 710 is set, and the CONFIGURE_RECEIVED 

=13 20 indicator is reset 720. The QUIESCE messaqe is forwarded to 
downstream routing nodes 730 and the QUIESCE_CHILDREN count 
is set to the number of downstream routing nodes 740. If 
the QUIESCE_CHILDREN count is zero, then the QUIESCE_ACK 
message is returned to the node's parent node 750. 

25 One embodiment for processing (pursuant to the present 

invention) a QUIESCE_ACK message is depicted in Fig. 8. 
Upon receipt of a QUIESCE_ACK message 800 at a node, the 
node's QUIESCING_CHILDREN count is decremented 810 and the 
node determines whether the QUIESCING CHILDREN count is zero 
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820. If ''no", then (in this embodiment) the node awaits 
further QUIESCE_ACK messages, meaning that the current 
processing is done 830. 

Once all children have responded with a QUIESCE_ACK 
5 message, the node determines whether it is the 

reconfiguration root 840. If ''no", the node sends a 
QUIESCE_ACK to it parent node 845. If the node is the root 
node for the reconfiguration, then the node sends a 
QUIESCE_SPECIAL message to its children nodes having a 

10 special node in its sub-tree 850. Also, note from Fig. 8, 
that if the QUIESCING__CHILDREN count is zero in step 680 of 
Fig. 6, then processing jumps to instruction 850, After 
sending the QUIESCE_SPECIAL message, the node sets the 
QUIESCING_CHILDREN count to the number of children to which 

15 the QUIESCE_SPECIAL message was sent 8 60. If the 

QUIESCING_CHILDREN count equals zero 870, then processing 
goes to step 1050 of Fig. 10. 

Fig. 9 depicts a node's processing of a QUIESCE_SPECIAL 
message in accordance with the present invention. After a 

20 QUIESCE_SPECIAL message is received at a node 900, the node 
forwards the message to appropriate children nodes if a 
special node exists in a sub-tree from that node 910. The 
QUIESCING_CHILDREN indicator is set to the number of 
children nodes to which the QUIESCE_SPECIAL message is 

25 forwarded 920. If the QUIESCING_CHILDREN indicator equals 
zero, then a QUIESCE_SPECIAL_ACK message is returned to the 
node's parent node 930. 

One embodiment for processing the QUIESCE__SPECIAL_ACK 
message is depicted in Fig. 10. Upon receipt of a 
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QUIESCE_SPECIAL_ACK message at a node 1000, the node 
decrements its QUIESCING_CHILDREN count 1010, and determines 
whether the QUIESCING_CHILDREN count is zero 1020. If ''no", 
this processing is done 1025. Otherwise, the node 
5 determines whether it is the root node for the 

reconfiguration 1030. If ''no", then a QUIESCE_SPECIAL_ACK 
message is sent to its parent node 1040. 

If the root comprises the reconfiguration root, or if 
the node's QUIESCING_CHILDREN count equals zero (step 870 of 
10 Fig. 8), processing sends a QUIESCE_COMPLETE message to the 
node's children in the old configuration 1050. In addition, 
a CONFIGURE message is sent to the children in the new 
■13 configuration 1060, and the RECONFIGURE_ROOT and QUIESCING 

indicators are reset 1070. The node then processes any 
ly 15 messages in the QUIESCE_MSG_QUEUE 1080. 

Fig. 11 depicts one embodiment for processing a 
JL QUIESCE_COMPLETE message in accordance with the principles 

y of the present invention. Upon a node's receipt of a 

Jf:? QUIESCE_COMPLETE message 1100, the node resets the QUIESCING 

y1 20 indicator 1110. The QUIESCE_COMPLETE message is then 

forwarded to its children nodes with respect to the current 
spanning tree 1120. The node inquires whether the 
CONFIGURE_RECEIVED indicator is set 1130 and, if not, 
processing is completed 1140. Otherwise, the 
25 reconfiguration is complete and the node begins forwarding 
any messages from the QUIESCE_MSG_QUEUE using the new 
spanning tree 1150. 

Fig. 12 depicts one embodiment for processing published 
messages in accordance with the present invention. Upon 
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receipt of a published message 1200, a node determines 
whether the QUIESCING indicator is set 1210. If ^^no'^ then 
the message is processed normally 1220, If in quiescing 
state, and the message is received from a local client or 
5 the current node is the reconfiguration root node and the 

message is received from a parent node, the message is added 
to the node's QUIESCE_MSG_QUEUE 1250/ Otherwise, the 
message is processed normally 1240. 

Those skilled in the art will note from the description 

10 provided herein that if configuration changes are confined 
to a sub-tree of the routing network, it is possible to 
confine the above-described processing to that sub-tree. In 
such a case, the root is the parent node of the sub-tree 
under which all configuration changes occur. If the sub- 

15 tree does not contain a logger/sequencer, it is not 

necessary to quiesce the logger/sequencers. In such a case, 
the control message quiescence phase may be skipped. If the 
sub-tree does contain logger/sequencer node(s), the 
logger/sequencer nodes within the sub-tree must be quiesced. 

20 So, the control message quiescence phase would be performed. 
Unlike the above description, data messages may be sent to 
the root by the parent of the root after data messages have 
been quiesced in the sub-tree. In this case, these data 
messages should be queued at either the root of the sub- 

25 tree, or at the logger/sequencer nodes (this would only be 
possible if a logger/sequencer maintained the same function 
in the new configuration) . Logging acknowledgment or 
sequencing acknowledgment messages must not be sent in 
response to these data messages until after the network has 

30 been reconfigured. 
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Also, those skilled in the art will note from the 
description provided herein that if there are multiple 
spanning trees within the routing network and if 
configuration changes are required to more than one spanning 
tree, it is possible to extend the above-described 
processing to effect these changes simultaneously. 

The present invention can be included, for example, in 
an article of manufacture (e.g., one or more computer 
program products) having, for instance, computer usable 
media. This media has embodied therein, for instance, 
computer readable program code means for providing and 
facilitating the capabilities of the present invention. The 
articles of manufacture can be included as part of the 
computer system or sold separately. 

Additionally, at least one program storage device 
readable by machine, tangibly embodying at least one program 
of instructions executable by the machine, to perform the 
capabilities of the present invention, can be provided. 

The flow diagrams depicted herein are provided by way 
of example. There may be variations to these diagrams or 
the steps (or operations) described herein without departing 
from the spirit of the invention. For instance, in certain 
cases, the steps may be performed in differing order, or 
steps may be added, deleted or modified. All of these 
variations are considered to comprise part of the present 
invention as recited in the appended claims. 

While the invention has been described in detail herein 
in accordance with certain preferred embodiments thereof. 
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many modifications and changes therein may be effected by 
those skilled in the art. Accordingly, it is intended by 
the appended claims to cover all such modifications and 
changes as fall within the true spirit and scope of the 
5 invention . 
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Claims 



1 1. A method for reconfiguring a routing network 

2 comprising : 

3 quiescing the routing network to preserve a first - 

4 in first -out ordering of data messages within the 

5 routing network; and 

6 reconfiguring the quiesced routing network. 

1 2. The method of claim 1, wherein said quiescing 

2 further comprising quiescing control messages within said 

3 routing network prior to said reconfiguring. 

1 3. The method of claim 2, wherein said routing 

2 network comprises at least one of a logging network or a 

3 sequencing network, said logging network including logging a 

4 message to persistent storage within the routing network 

5 prior to delivery of the message to one or more clients of 

6 the network, and said sequencing network comprising 

7 sequencing a message at a node of the routing network with 

8 other messages received by the network prior to delivery of 

9 the message to one or more clients of the network, wherein 

10 said quiescing of said control mess.ages comprises quiescing 

11 any new logging acknowledgment message or sequencing 

12 acknowledgment message within said logging network or 

13 sequencing network, respectively, 

1 4, The method of claim 1, wherein said quiescing 

2 comprises sending a quiesce message from a root node to all 

3 children nodes thereof using an existing routing network 

4 configuration. 
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1 5. The method of claim 4, wherein when a node of said 

2 routing network receives the quiesce message, the node 

3 begins queuing any new data messages received from a client 

4 the reof , forwards the quiesce message to any children nodes 



5 thereof, and sends a quiesce acknowledgment to its parent 

6 node, and wherein once the root node has received a quiesce 

7 acknowledgment from all children nodes, all data messages in 

8 a sub- tree defined from the root node have been quiesced. 



1 6, The method of claim 1, wherein said routing 

2 network is part of a publish/subscribe system, and wherein 

3 said reconfiguring is transparent to both publishers and 

4 subscribers of said publish/subscribe system. 

1 7. The method of claim 1, wherein said routing 

2 network comprises a continuously available broker network. 

1 8. The method of claim 1, wherein said quiescing 

2 comprises propagating a quiesce message from a root node 

3 across a spanning tree of said routing network, said quiesce 

4 message preventing any new data messages from being 

5 published for routing across said spanning tree. 

1 9, The method of claim 8, wherein each node of said 

2 spanning tree responds to said quiesce message by sending a 

3 quiesce acknowledgment message to its parent node, and 

4 wherein after sending said quiesce acknowledgment message, 

5 any new data message received from a client of said node is 

6 queued at said node. 

1 10. The method of claim 9, further comprising 

2 quiescing control messages within said routing network prior 

3 to said reconfiguring. 
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1 11. The method of claim 10, wherein said quiescing of 

2 control messages comprises sending' a quiesce special message 

3 from said root node to a special node of said routing 

4 network, said special node of said routing network 

5 comprising one of a logger node or a sequencer node. 

1 12. The method of claim 11, wherein upon receipt of 

2 said quiesce special message at said special node, said 

3 special node returns a quiesce special acknowledgment 

4 message to said root node, and wherein upon receipt of said 

5 root node of said quiesce special acknowledgment message, 

6 said control messages within said routing network have been 

7 quiesced. 

1 13. The method of claim 9, wherein said reconfiguring 

2 comprises sending a quiesce complete message from said root 

3 node to nodes of said spanning tree, wherein said spanning 

4 tree comprises an existing spanning tree configuration, and 

5 then sending a configure message to said nodes using a new 

6 spanning tree configuration, thereby accomplishing said 

7 reconfiguration . 

1 14. The method of claim 1, wherein said quiescing 

2 further comprises quiescing control messages within said 

3 routing network prior to said reconfiguring, and said 

4 reconfiguring comprises reconfiguring said quiesced routing 

5 network transparent to any clients of said routing network, 

6 and wherein said routing network comprises one of a logging 

7 routing network wherein messages are logged for guaranteed 

8 delivery, or a sequencing routing network wherein messages 

9 are sequenced for ordered delivery. 
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1 15, The method of claim 14, wherein said network is 

2 part of a publish/subscribe system supporting content-based 

3 subscription, and wherein said method is used in combination 

4 with a method for routing messages within said routing 

5 network, said method for routing comprising routing a 

6 message to one or more clients of said network, said routing 

7 being based on data content of said message irrespective of 

8 any destination information that may be within said message. 

1 16, The method of claim 15, wherein said routing 

2 comprises logging said message at at least one logging node 

3 within said network before delivering said message to said 

4 one or more clients of said network, said logging comprising 

5 storing said message in persistent storage. 

1 17, The method of claim 14, further in combination 

2 with a method for routing messages within the routing 

3 network, said routing method comprising sequencing a message 

4 at a node of the routing network with other messages 

5 received by the routing network to produce an ordering of 

6 message delivery within the routing network, and delivering 

7 the message to one or more clients of the routing network 

8 while maintaining the ordering of message delivery, 

1 18. The method of claim 17, wherein said network is 

2 part of a publish/subscribe system. 
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19, A system for reconfiguring a routing network 
comprising: 



3 means for quiescing the routing network to 

4 preserve a first-in first-out ordering of data messages 

5 within the routing network; and 

6 means for reconfiguring the quiesced routing 

7 network , 

1 20, The system of claim 19, further comprising means 

2 for quiescing control messages within said routing network 

3 prior to reconfiguration thereof. 

1 21. The system of claim 20, wherein said routing 

2 network comprises at least one of a logging network or a 

3 sequencing network, said logging network including a special 

4 logger node for logging a message to persistent storage 

5 within the routing network prior to delivery of the message 

6 to one or more clients of the network, and said sequencing 

7 network comprising a special sequencing node for sequencing 

8 a message within the routing network with other messages 

9 received by the network prior to delivery of the message to 

10 one or more clients of the network, wherein the means for 

11 quiescing of the control messages comprises means for 

12 quiescing any new logging acknowledgment message or 

13 sequencing acknowledgment message within the logging network 

14 or sequencing network, respectively, 

1 22. The system of claim 19, wherein said means for 

2 quiescing comprises means for sending a quiesce message from 

3 a root node to all children nodes thereof using an existing 

4 routing network configuration. 
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1 23. The system of claim 22, further comprising means 

2 for queueing any new data messages received from a client of 

3 a node after the node receives the quiesce message, and for 

4 forwarding the quiesce message to any children nodes of said 

5 node, and for sending a quiesce acknowledgment to its parent 

6 node, wherein once the root node has received a quiesce 

7 acknowledgment from all children nodes, all data messages in 

8 a sub- tree defined from the root node have been quiesced, 

1 24. The system of claim 19, wherein said routing 

2 network is part of a publish/subscribe system, and wherein 

3 said means for reconfiguring comprises means for 

4 reconfiguring the quiesced routing network transparent to 

5 both publishers and subscribers of said publish/subscribe 

6 system. 

1 25. The system of claim 19, wherein said routing 

2 network comprises a continuously available broker network. 

1 26. The system of claim 19, wherein said means for 

2 quiescing comprises means for propagating a quiesce message 

3 from a root node across a spanning tree of said routing 

4 network, said quiesce message preventing any new data 

5 messages from being published for routing across said 

6 spanning tree . 

1 27. The system of claim 26, further comprising means, 

2 at nodes of said spanning tree, for sending a quiesce 

3 acknowledgment message to its parent node in response to 

4 receipt of said quiesce message, and for queueing any new 

5 data message received from a client of said node after 

6 sending said quiesce acknowledgment message. 
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1 28. The system of claim 27, further comprising means 

2 for quiescing control messages within said routing network 

3 prior to said reconfiguration by said means for 

4 reconfiguring . 

1 29. The system of claim 28, wherein said means for 

2 quiescing control messages comprises means for sending a 

3 quiesce special message from said root node to a special 

4 node of said routing network, said special node of said 

5 routing network comprising one of a logger node or a 

6 sequencer node. 

1 30. The system of claim 29, further comprising means 

2 for returning a quiesce special acknowledgment message from 

3 said special node upon receipt of said quiesce special 

4 message, wherein said routing network has been quiesced upon 

5 receipt at said root node of said quiesce special 

6 acknowledgment message. 

1 31. The system of claim 27, wherein said means for 

2 reconfiguring comprises means for sending a quiesce complete 

3 message from said root node to nodes of said spanning tree, 

4 wherein said spanning tree comprises an existing spanning 

5 tree configuration, and thereafter, for sending a configure 

6 message to nodes of a new spanning tree configuration, 

7 thereby accomplishing said reconfiguration. 
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1 32. The system of claim 19, wherein said means for 

2 quiescing further comprises means for quiescing control 

3 messages within said routing network prior to said 

4 reconfiguration by said means for reconfiguring, and said 

5 means for reconfiguring comprises means for reconfiguring 

6 said quiesced routing network transparent to any clients of 

7 said routing network, and wherein said routing network 

8 comprises one of a logging network wherein messages are 

9 logged for reliable routing, or a sequencing network wherein 
10 messages are sequenced for ordered delivery. 

1 33. The system of claim 32, wherein said network is 

2 part of a publish/subscribe system supporting content-based 

3 subscription, and wherein said system is used in combination 

4 with a system for routing messages within said routing 

5 network, said system for routing messages comprising means 

6 for routing a message to one or more clients of said 

7 network, said means for routing being based on data content 

8 of said message irrespective of any destination information 

9 within said message. 

1 34, The system of claim 33, wherein said means for 

2 routing comprises means for logging said message at at least 

3 one logging node of said network before delivering said 

4 message to said one or more clients of said network, said 

5 means for logging comprising means for storing said message 

6 in persistent storage. 
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1 35. The system of claim 32, further in combination 

2 with a system for routing messages within the routing 

3 network, said routing system comprising means for sequencing 

4 a message at a node of the routing network with other 

5 messages received by the routing network to produce an 

6 ordering of message delivery within the routing network, and 

7 means for delivering the message to one or more clients of 

8 the routing network while maintaining the ordering of 

9 message delivery. 

1 36. The system of claim 35, wherein said routing 

2 network is part of a publish/subscribe system. 
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1 37. A system for reconfiguring a publish/subscribe 

2 system comprising: 

3 a routing network adapted to receive published 

4 messages for forwarding to subscribers; and 

5 said network being further adapted to quiesce data 

6 messages and control messages within said network in 

7 response to a reconfigure command to preserve a first - 

8 in first-out ordering of data and control messages 

9 within the routing network, and to then reconfigure the 

10 routing network once data messages and control messages 

11 have been quiesced. 
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38. An article of manufacture comprising: 



2 at least one computer usable medium having 

3 computer readable program code means embodied therein 

4 for effecting reconfiguring of a routing network, the 

5 computer readable program code means in the article of 

6 manufacture comprising: 

7 computer readable program code means for 

8 causing a computer to effect quiescing the routing 

9 network to preserve a first-in first -out ordering 

10 of data messages within the routing network; and 

11 computer readable program code means for 

12 causing a computer to effect reconfiguring the 

13 / quiesced routing network. 

1 39. The article of manufacture of claim 38, wherein 

2 said routing network is part of a publish/subscribe system. 

1 40. The article of manufacture of claim 39, wherein 

2 said publish/subscribe system includes at least one special 

3 node within said routing network, said at least one special 

4 node comprising one of a logging node or a sequencing node, 

5 wherein said logging node is employed to log messages to 

6 persistent storage, and said sequencing node is employed to 

7 sequence messages for ordered delivery. 
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1 41. The article of manufacture of claim 40, wherein 

2 said computer readable program code means for causing a 

3 computer to effect reconfiguring comprises computer readable 

4 program code means for causing a computer to effect 

5 reconfiguring the quiesced routing network transparent to 

6 both publishers and subscribers of said publish/subscribe 

7 system. 
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QUIESCENT RECONFIGURATION OF 
A ROUTING NETWORK 

Abstract of the Disclosure 

Reconfiguration of a continuously available routing 
5 network of a publish/subscribe system is achieved by 

quiescing data messages and control messages within the 
routing network to preserve first -in first -out ordering 
thereof, and then, reconfiguring the quiesced routing 
network. This reconfiguration technique is particularly 

10 applicable to a routing network having at least one special 
node comprising one of a logger node or a sequencer node, 
wherein the logger node is used to store a message to 
persistent storage within the routing network prior to 
delivery of the message to one or more clients, and the 

15 sequencer sequences a message with other messages received 
by the network prior to delivery thereof for ordered 
delivery of the messages to one or more clients of the 
network. 
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Declaration and Power of Attorney For Patent Application 

English Language Declaration 

As a below named inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated below next to my name, 

I believe I am the original, first and sole inventor (if only one name is listed below) or an original, 
first and joint inventor (if plural names are listed below) of the subject matter which is claimed and for 
which a patent is sought on the invention entitled 

QUIESCENT RECONFIGURATION OF A ROUTING NETWORK 

the specification of which 

O (check one) 

jij D is attached hereto. 

B ° - States Application No. or PCT International 

jl Application Number 



and was amended on 



(if applicable) 

jjj I hereby state that I have reviewed and understand the contents of the above identified specification, 
U '"^'"^ding the claims, as amended by any amendment referred to above. 

~q\ acknowledge the duty to disclose to the United States Patent and Trademark Office all information 
di known to me to be material to patentability as defined in Title 37, Code of Federal Regulations 
Section 1.56. 



I hereby claim foreign priority benefits under Title 35, United States Code, Section 119(a)-(d) or 
Section 365(b) of any foreign application (s) for patent or inventor's certificate, or Section 365(a) of 
any PCT International application which designated at least one country other than the United 
States, listed below and have also identified below, by checking the box, any foreign application for 
patent or inventor's certificate or PCT International application having a filing date before that of the 
application on which priority is claimed. 

Prior Foreign Application(s) Priority Not Claimed 



— □ 

(Number) (Country) (Day/MonthA'ear Filed) 

— . □ 

(Number) (Country) (Day/MonthA'ear Filed) 

□ 

(Number) (Country) (Day/MonthA'ear Filed) 
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I hereby claim the benefit under 35 U.S.C. Section 119(e) of any United States provisional 
application(s) listed below: 



(Application Serial No.) 


(Filing Date) 


(Application Serial No.) 


(Filing Date) 



(Application Serial No.) (Filing Date) 



I liereby claim the benefit under 35 U. S. C. Section 120 of any United States application(s), or 
Section 365(c) of any PCT International application designating the United States, listed below and, 
insofar as the subject matter of each of the claims of this application is not disclosed in the prior 
United States or PCT International application in the manner provided by the first paragraph of 35 
U.S.C. Section 112, I acknowledge the duty to disclose to the United States Patent and Trademark 
O Office all information known to me to be material to patentability as defined in Title 37, C. F. R., 
jO Section 1.56 which became available between the filing date of the prior application and the national 
^ or PCT International filing date of this application: 

fr- 1 



:j (Application Serial No.) 


(Filing Date) 


(Status) 

(patented, pending, abandoned) 


jj (Application Serial No.) 


(Filing Date) 


(Status) 

(patented, pending, abandoned) 


|1 (Application Serial No.) 


(Filing Date) 


(Status) 

(patented, pending, abandoned) 



I hereby declare that all statements made herein of my own knowledge are true and that all 
statements made on information and belief are believed to be true; and further that these statements 
were made with the knowledge that willful false statements and the like so made are punishable by 
fine or imprisonment, or both, under Section 1001 of Title 18 of the United States Code and that 
such willful false statements may jeopardize the validity of the application or any patent issued 
thereon. 
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POWER OF ATTORNEY: As a named inventor, I hereby appoint the following attorney(s) and/or 
agent(s) to prosecute this application and transact all business in the Patent and Trademark Office 
connected therewith, (list name and registration nun)ber) 



Manny W, Schecter - Reg. 31,722 
Terry J. Dardi - Reg. No. 29,936 
Christoper A. Hughes - Reg. No. 26,914 
Edward A. Pennington - Reg. No. 32,588 
John E. Hoel - Reg. No. 26,279 
Joseph C. Redmond, Jr. - Reg. No. 18,753 
Kevin M. Jordan - Reg. No. 40,277 
Stephen C. Kaufman - Reg. No. 29,551 
Jay P. SbroUini - Reg. No. 36,266 



David M. Shofi - Reg. No. 39,835 
Robert M. Trepp - Reg. No. 25,933 
Louis P. Herzberg - Reg. No. 41,500 
Daniel P. Morris - Reg. No. 32,053 
Douglas W. Cameron - Reg. No. 31,596 
Paul J. Otterstedt - Reg. No. 37,411 



Send Correspondence to: ^* Radigan, Esq. 

HESLIN & ROTHENBERG, P.C. 
5 Columbia Circle 
% Albany, NY 12203 



J Direct Telephone Calls to: (name and telephone number) 




% Kevin P. Radigan, Esq. - Tel.: (518) 452-5600 






[ Full name or sole or first inventor 




Kevan Lee Miller 




Sole or first inventor's signature 


Date 


Residence 




J 3 Kilian Drive, Danbury, CT 06811 




,F Citizenship 




United States of America 




J Post Office Address 




3 Kilian Drive 




Danbury, CT 06811 



Full name of second inventor, If any 




Robert Evan Strom 




Second inventor's signature 


Date 


Residence 




6 Rochambeau Ave., Ridgefield, CT 06877 




Citizenship 




United States of America 




Post Office Address 




6 Rochambeau Ave. 




Ridgefield, CT 06877 
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h uii name or third inventor, if any ~ ' 

Daniel Charles Sturman 






1 nird inventors signature 


Date 






Residence ~ " — " ~ 

408 Knickerbocker Road, Englewood, NJ 07631 






Citizenship ~ ~ ~ ' " ~~ 

United States of America 






Post Office Address " ~ ~ ■ 

408 Knickerbocker Road 






EngIewood,NJ 07631 












Full name of fourth inventor, if any ~ ' ' ■ 

Michael James Ward 






Fourth inventor's signature 


Date 






Residence 

25 West Park Avenue, New Haven, CT 06511 




Hi 


Citizenship ' ^ ' ~ 

United States of America 




I* 


Post Office Address ~ ~~~ ~ ' 

25 West Park Avenue 






New Haven, CT 06511 
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Full name of fifth inventor, if any ' ~ " " 






Finn inventors signature 


Date 






Residence 




^£ 


Citizenship ~ — 






Post Office Address ' ~ 


















Full name of sixth inventor, if any 






Sixth inventors signature 


Date 






Residence 






Citizenship 






Post Office Address 
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